Skip to content

feat: Add Apify adapter for social media monitoring#89

Merged
boringdata merged 19 commits intokurt-simplificationfrom
apify-integration-migration
Jan 28, 2026
Merged

feat: Add Apify adapter for social media monitoring#89
boringdata merged 19 commits intokurt-simplificationfrom
apify-integration-migration

Conversation

@boringdata
Copy link
Owner

Summary

Migrates the Apify integration from PR #81 to follow the new patterns in the kurt-simplification branch.

  • ApifyAdapter - Adapter for Twitter/X, LinkedIn, and Threads via Apify actors
  • Actor Registry - Built-in presets for common platforms with custom actor support
  • Flexible Field Mapping - Extract fields with fallback lists or callables
  • CLI Support - kurt integrations research onboard --source apify
  • 22 new tests - Full test coverage for the adapter

Files Changed

File Description
monitoring/apify.py New ApifyAdapter (635 lines)
monitoring/__init__.py Export ApifyAdapter
config.py Add Apify config template
cli.py Add Apify onboard/status support
tests/test_adapters.py Add 22 ApifyAdapter tests

Usage

from kurt.integrations.research.monitoring import ApifyAdapter

adapter = ApifyAdapter({"api_token": "apify_xxx"})

# High-level API
signals = adapter.search_twitter("AI agents", max_items=50)
signals = adapter.search_linkedin("B2B marketing")

# Mid-level API
signals = adapter.fetch_signals("query", actor="apidojo/tweet-scraper")

# Low-level API
items = adapter.run_actor("any/actor", {"custom": "input"})
signals = adapter.parse_results(items, source="custom")

Test plan

  • All 22 new ApifyAdapter tests pass
  • All existing research adapter tests pass (32 total)
  • Manual test with real Apify token

🤖 Generated with Claude Code

Kurt User and others added 19 commits January 27, 2026 19:43
- Add check_cancel_requested(run_id) to poll for cancel status
  - Returns True when workflow status is 'canceling'
  - Workflows should call this periodically in their main loop

- Add on_workflow_cancel(run_id, message) to complete cancellation
  - Validates workflow is in 'canceling' status
  - Updates status to 'canceled' with timestamp
  - Emits cancellation event

This completes the cancel flow:
1. Frontend calls POST /api/workflows/{id}/cancel
2. API sets status to 'canceling'
3. Workflow calls check_cancel_requested() periodically
4. When True, workflow cleans up and calls on_workflow_cancel()
5. Status becomes 'canceled'

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Search now matches workflow name in addition to ID
- Add has_more, offset, limit fields to pagination response
- Track raw_count before Python filtering for accurate has_more

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Tool workflows (map, fetch, signals, etc.) now set workflow_type="tool"
- TOML workflows set workflow_type="toml"
- Agent workflows already set workflow_type="agent"
- Also propagate parent_workflow_id for nested workflow tracking

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- /api/workflows/{id}/logs/stream now streams from step_events table
- /api/workflows/{id}/logs returns structured events from step_events
- Both endpoints also support file-based logs as fallback
- Add step_id filtering parameter to both endpoints
- Return structured event data with metadata parsing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Store parent_step_name in child workflow metadata
- Read from KURT_PARENT_STEP_NAME environment variable
- Set KURT_PARENT_STEP_NAME="agent_execution" for agent subprocesses
- Return parent_step_name in workflow list API response
- Enables frontend to group child workflows by parent step

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Store token counts and cost in workflow metadata on completion (#13)
- Add step_type="queue" to agent step logs for frontend display (#21)
- Enables frontend to show costs and identify queue steps

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Disable link and underline in StarterKit since we configure them separately
- Fixes console warnings about duplicate extension names

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add _denormalize_status_filter() to convert frontend status filters
  (SUCCESS, ERROR, PENDING, CANCELLED) to database values
- Use IN clause for PENDING filter which maps to multiple DB states
  (pending, running, canceling)
- Remove pytest-postgresql from dev dependencies (not needed)
- Update uv.lock to reflect dependency changes

Fixes workflow list filtering which was broken because the UI sends
uppercase statuses but the database stores lowercase values.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add _normalize_step_status() to convert step/event status to frontend format
  (completed->success, failed->error, progress->progress)
- Apply normalization to:
  - /api/workflows/{id}/step-logs endpoint
  - /api/workflows/{id}/logs endpoint
  - /api/workflows/{id}/logs/stream SSE events
- Normalize workflow status in stream done message

This ensures the frontend receives consistent status values from all
endpoints instead of mixing raw DB values with normalized values.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Auto-expand on error (#20):
- Add hasErrorStatus() helper to detect workflows with errors
- Track auto-expanded and user-collapsed workflows with refs
- Automatically expand first error workflow on load
- Respect user's manual collapse (don't re-expand)

Polling optimization (#19):
- Add adaptive polling intervals (2s fast, 10s slow)
- Poll fast when workflows are actively running (PENDING/ENQUEUED)
- Poll slow when all workflows are idle/completed
- Split useEffect for cleaner separation of concerns

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Output/Result display (#28):
- Add _build_output_summary() to extract output metrics
- Show agent metrics: turns, tokens, cost, tool_calls
- Show tool metrics: output_count, success, errors
- Display result preview and errors prominently
- Auto-expand output section when errors present

Retry functionality (#26):
- Add POST /api/workflows/{id}/retry endpoint
- Handle both agent and tool workflow retries
- Preserve original inputs for retry
- Add retry button in UI for completed/failed workflows

Config/Definition display (#27):
- Add WorkflowConfigSection component
- Show workflow_type, definition_name, trigger
- Display inputs in formatted key-value grid
- Collapsible section with smart preview

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…detail panel

- Add WorkflowTimeline.jsx: Horizontal bar chart visualization of step execution
  - Color-coded status (green=success, red=error, blue=running, gray=pending)
  - Duration bars proportional to execution time
  - Hover tooltips with step details

- Add WorkflowMetrics.jsx: Aggregate statistics dashboard
  - Total workflows by status (success/error/running counts)
  - Total cost and tokens across all workflows
  - Average duration

- Add WorkflowDetailPanel.jsx: Dedicated workflow detail view
  - Full workflow metadata and inputs display
  - Integrated timeline visualization
  - Expandable step cards with error details
  - Action buttons (cancel, retry, attach terminal)
  - Auto-refresh toggle with adaptive polling

- Update CLAUDE.md: Document Workflow Observability API
  - Table schemas (workflow_runs, step_logs, step_events)
  - All API endpoints with request/response examples
  - Status values and transitions
  - Programmatic usage examples

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add expandable step logs in timeline: click any step bar to see its events
  - Shows event timestamp, status icon, message, and progress
  - Fetches from /api/workflows/{id}/logs?step_id=<step>
  - Lazy loading - only fetches when step is expanded

- Distinct visual styles for Config and Output sections:
  - Config: Blue left border, settings icon (⚙), blue-tinted header
  - Output: Green left border, chart icon (📊), green-tinted header
  - Error Output: Red left border and error styling

- Timeline improvements:
  - Step labels are now clickable with expand arrow icon
  - Step bars are clickable to toggle expansion
  - Smooth rotation animation on expand/collapse

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migrate Apify integration from PR #81 to follow the new patterns in
kurt-simplification branch. The ApifyAdapter provides access to Twitter/X,
LinkedIn, and Threads via Apify actors.

Features:
- Actor registry with built-in presets for common platforms
- Flexible field mapping for custom actors
- Three API levels: high-level (search_twitter), mid-level (fetch_signals),
  low-level (run_actor + parse_results)
- Config via kurt.config (RESEARCH_APIFY_API_TOKEN)
- CLI support: kurt integrations research onboard --source apify

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… block

- Remove separate WorkflowTimeline component from main workflow view
- Integrate timeline bar directly into StepBox - shows when step is expanded
- Add step events display within StepBox (fetched from /logs endpoint)
- Progressive disclosure: Steps header → click step → see bar + events

Visual structure now:
  Steps (6 processed)
    └─ agent_execution (6 ok, 0 errors, 41.0s)
       └─ [expanded: timeline bar + events list]

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Enhance track-tool hook to capture input/output summaries for each tool call
- Store tool calls as events in database (not just count)
- Replace empty legacy logs box with "Details" section showing:
  - Tool, Model, Tokens In/Out, Cost, Stop Reason
- Tool calls will appear as events when workflows are run with new code

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update track-tool hook to write directly to step_events table
  instead of temp file (enables real-time monitoring)
- Add _get_kurt_executable() to resolve absolute path for hooks
- Change hook command to "kurt workflow track-tool" for proper routing
- Update WorkflowRow.jsx to display tool call events with details
- Add comprehensive unit tests for tool tracking functionality:
  - Test DB writes with workflow ID
  - Test input summary extraction for 7 tool types
  - Test metadata JSON format
  - Test graceful error handling

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove unused datetime import in agents/cli.py
- Remove unused pytest import in test_status.py
- Remove unused adapter variable in test_adapters.py
- Fix import sorting in dolt.py

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@boringdata boringdata merged commit 321d02b into kurt-simplification Jan 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant